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ABSTRACT 

The  Intelligent  Systems  And  Autonomous  Controls  (ISAAC)  robot 
is  an  experimental  autonomous  research  platform  being  developed  to 
advance  current  dismount  following  applications.  Specifically, 
vision  based  following  using  pedestrian  detection.  The  current 
standard  in  mule  applications  is  the  following  of  GPS  waypoints. 
ISAAC  is  designed  to  follow  a  specific  person  using  solely  vision 
based  techniques. 

The  core  of  the  vision  based  algorithms  used  in  this  application  is 
based  on  years  of  research  from  a  collaboration  of  government  and 
university  partners.  Stereo  vision  techniques  determine  a  person, 
identify  them  as  the  leader,  and  map  a  path  for  autonomous 
following. 


Ahead  Of  The  Pack 

In  order  to  keep  with  the  vision  of  a  more  agile  army,  tomorrow’s  soldier  needs  to  be  as 
unburdened  and  mobile  as  possible.  One  way  to  help  achieve  this  goal  is  to  alleviate  the 
individual  soldier  of  non-combat  essential  gear.  By  allowing  the  soldier  to  leave  this  gear 
on  some  sort  of  transport,  the  soldier  can  march  “ahead  of  the  pack”  and  increase  their 
individual  combat  effectiveness. 

An  integral  part  of  this  objective,  and  the  future  force  itself,  is  the  FCS  MULE  (The 
Multifunction  Utility/Logistics  Equipment  Vehicle).  The  MULE  is  intended  to  be  an 
unmanned  platform  capable  of  providing  transport  for  equipment  and  supplies  in  support 
of  dismounted  infantry.  Anything,  that  is  essential  to  the  mission,  but  not  to  the 
individual  soldier,  can  be  placed  on  the  MULE.  A  similar  varient  (ARV- Assault  (Light) 
will  be  capable  of  carrying  armaments  to  neutralize  enemy  dismounted  infantry  as  well  as 
suppress  fortified  positions  at  a  tactical  assault  range.  Other  possible  capabilities  for 
growth  to  both  the  MULE  and  ARV-A(L)  include  acting  as  a  battery  charging  station  for 
future  Force  Warrior  (OFW)  personal  electronics  and  other  SUGVs,  water  generation  and 
purifications  systems,  and  advanced  sensor  suits  to  aid  the  soldiers’  vision  and  detect 
possible  threats  that  unaided  would  go  unnoticed  (e.g.  chemical  or  biological  threats.) 
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The  MULE  is  coming.  Multiple  variants  have  been  designed  and  are  being  tested.  Many 
different  methods  for  the  soldier’s  control  of  the  vehicle  are  being  investigated.  One  such 
method  is  the  Leader-Follower  approach. 

In  the  Field 

So  what  is  the  current  state  of  the  art  for  leader- 
follower  applications?  One  of  the  leaders  in  this  field 
is  the  RF  ATD  (Robotic  Follower  Advanced 
Technology  Demonstration)  program  managed  at  the 
U.  S.  Army  TARDEC  (Tank- automotive  and 
Armaments  Research  Development  and  Engineering 
Center).  Under  this  program,  they  recently  completed 
experiments  using  a  MULE  class  chassis.  Building 
upon  results  from  the  GPS  based  following  done  at 
Fort  Bliss  in  March  of  2003,  they've  advanced  the  technology  in  the  following  ways. 

First,  they  refined  their  leader  localization  accuracy  through  Human  Pedometer 
technology  for  dismount  navigation  to  compensate  for  GPS  dropout.  Two  approaches 
were  developed  for  this;  one  by  the  University  of  Michigan  and  the  other  by  Carnegie 
Mellon  University.  CMU’s  approach  was  tested  for  accuracy  at  Fort  Indiantown  Gap  in 
Dec  of  2004  and  yielded  very  promising  results.  Second,  the  hardware  the  leader  is 
required  to  carry  has  been  reduced  and  ruggedized  with  respects  to  size  and  power.  The 
current  vehicle  has  been  proven  to  follow  a  dismounted  leader  with  separation  distances 
as  little  as  20  m  and  time  delays  up  to  6  hours  over  various  terrains  from  urban  to  desert. 

Another  up-and-coming  leader  in  this  field  is  the  joint  effort 
between  iRobot  and  John  Derre.  The  Military  Robotic  Gator 
(R-Gator)  project  is  unique  because  it  will  be  the  first  COTS 
(commercial  off  the  shelf)  vehicle  for  the  robotic  world.  To 
date,  the  other  vehicles  of  this  size  with  comparable  capability 
have  all  been  custom-made.  The  R-Gator  is  capable  of  three 
modes  of  operation:  manual,  tele-op,  or  semi-autonomous.  It 
can  be  used  in  dismount  applications  by  being  directed  to 
follow  soldiers  or  a  set  route  using  GPS  waypoints.  It  could  also  be  used  in  autonomous 
convoying,  automatically  shuttling  between  rear  supply  points  and  forward  operating 
positions  by  using  either  manually  entered  waypoints  or  those  collected  dynamically 
during  operations.  The  first  R-Gators  are  scheduled  to  roll  off  the  assembly  line  by  mid- 
2005.  Full  production  is  planned  to  begin  in  2006. 

What  do  these  systems  have  in  common?  Both  of  these  platforms  are  representative  of 
the  state-of-the-art  of  current  leader-follower  technology  being  tested  by  the  US  ARMY. 
Both  have  been  tested  extensively  in  the  field  on  proven  platforms.  And  both  have  a 
heavy  dependence  on  GPS.  A  proven  technology  with  proven  limitations  in  the  types  of 
environments  a  dismounted  soldier  is  known  to  operate. 
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In  the  Footsteps  of  a  Soldier 

In  a  theater  of  operations  a  typical  war  fighter  can  find  themselves  in  many  hostile 
environments.  A  large  percentage  of  these  environments  are  not  very  GPS  friendly  and 
many  are  down  right  intolerable.  Since  GPS  receivers  use  signals  from  satellites  they 
must  have  a  clear  view  of  the  sky  at  all  times.  In  proximity  to  tall  buildings  or  in  dense 
forest  satellite  signal  may  be  poor  or  nonexistent.  In  order  to  get  a  three-dimensional 
position  (latitude,  longitude  and  height)  the  system  requires  a  lock  with  a  minimum  of 
four  satellites;  imposing  an  even  greater  cover  constraint.  Also,  because  GPS  is  based  on 
RF  technology,  anything  that  interferes  with  radio  communications  will  interfere  with 
GPS  more  so,  because  of  the  lower  power  and  longer  range  GPS  signals  must  travel. 
Finally,  because  most  leader-follower  applications  require  more  resolution  than  standard 
GPS  delivers;  augmented  GPS  is  required.  This  is  typically  thru  the  use  of  Differential 
GPS  (DGPS)  or  Wide  Area  Augmentation  System  (WAAS).  Both  of  which  require  an 
established  infrastructure. 

Now  let’s  put  this  into  historical  perspective.  Since  the  early  twentieth  century  the 
United  States  has  been  at  war  six  times:  World  War  I,  World  War  II,  the  Korean  War,  the 
Vietnam  War,  the  First  Persian  Gulf  War,  and  the  Iraq  War.  A  large  percentage  of  the 
conflicts  of  WWI  and  WWII  could  be  classified  as  Urban.  Where  as  both  the  Korean  and 
Vietnam  Wars  were  fought  under  the  canopy  deep  in  the  jungles  of  both  countries. 
Neither  of  these  environments  would  be  considered  ideal  for  GPS  operation.  This  just 
leaves  the  two  Gulf  Wars.  Therefore,  two  thirds  of  the  United  States  most  recent 
conflicts  were  in  regions  where  GPS  couldn’t  be  relied  upon.  And  while  the  open  deserts 
of  the  Gulf  Wars  is  an  ideal  environment  for  GPS  usage,  the  required  infrastructure  for 
higher  resolution  GPS  data  is  somewhat  lacking. 

Without  a  Trail 

As  was  previously  mentioned,  the  current  standard  in  mule  applications  is  the  following 
of  GPS  waypoints.  This  requires  a  considerable  amount  of  hardware  on  the  leader,  a 
dependence  on  GPS,  and  an  active  communications  link  between  the  leader  and  the 
vehicle.  None  of  these  are  tactically  desirable. 

However,  a  vision  based  approach  could  eliminate  all  these  shortcomings.  First,  a  fully 
developed  vision  based  system  would  not  require  the  leader  to  carry  any  additional 
hardware  to  interface  with  the  vehicle.  Second,  this  approach  doesn’t  use  GPS  and 
therefore  eliminates  this  dependency.  Finally,  because  there  is  no  additional  hardware 
requirement,  there  is  no  need  for  a  communications  link  to  the  vehicle.  This  eliminates 
the  threat  that  the  signal  is  detected  and/or  compromised. 

These  are  just  a  few  of  the  reasons  why  shifting  from  a  GPS-based  to  a  vision-based 
model  for  leader-follower  applications  would  be  highly  desirable.  By  reviewing  the 
current  GPS-based  model’s  limitations  this  should  have  become  more  apparent. 


Meeting  the  Need 
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For  the  past  year  and  half  TARDEC  has  had  two  In-house  Laboratory  Independent 
Research  (ILIR)  research  projects  that  have  focused  on  developing  technology  solutions 
for  this  very  problem. 

The  first  is  the  Line-of-sight  dismounted  leader-follower  capability  enhancements  using 
pedestrian  detection  ILIR.  The  objective  of  this  research  is  to  further  the  current 
developments  in  dismounted  leader-follower  technologies.  Recent  work  done  in  the  field 
of  leader-follower  development  has  not  been  focused  on  dismounted  applications.  This 
study  focuses  on  the  dismounted  soldier  while  leveraging  the  technologies  developed  for 
the  mounted  applications  -  specifically  research  in  the  field  of  pedestrian  detection.  The 
goal  is  to  refine  pedestrian  detection  algorithms  so  that  they  can  be  used  as  the  driving 
force  behind  dismounted  following. 

The  second,  begun  about  six  months  ago,  is  the  Pedestrian  Detection  ILIR.  The  goal  of 
this  research  is  detect  pedestrians  in  many  different  scenarios  for  autonomous  and  semi- 
autonomous  driving.  One  of  the  projects  under  the  Pedestrian  Detection  ILIR  is  human 
following.  The  goal  of  this  project  is  to  use  the  knowledge  of  the  other  pedestrian 
detection  projects  in  a  motion  based  following  capacity. 

ISAAC 

All  robotic  research  projects  need  a  test  bed,  a  platform  vehicle  that  can  be  used  prove 
out  the  technology  and  keep  the  developers  grounded  in  reality.  Theses  two  ILIRs  are  no 
different;  and  to  that  end  have  been  developing  ISAAC  (Intelligent  Systems  And 
Autonomous  Controls),  an  Unmanned  Ground  Vehicle  specifically  designed  for 
dismounted  following. 


ISAAC  was  designed  using  the  4D/RCS  reference 
architecture.  Use  of  this  framework  enabled  the 
merging  of  multiple  processors  and  systems  with 
varying  data  rates  as  well  as  giving  a  good  starting 
point  for  vehicle  design. 


The  vehicle  is  also  being  implemented  with  OE  3.0  and 
messaging  in  compliance  with  the  AS-4/JAUS 
Standard.  By  capitalizing  on  the  work  of  other  agencies 
and  development  teams  the  designers  hope  to  reduce 
the  risk  inherent  with  custom  platform  development  as 
well  as  speed  up  the  time  it  takes  to  field  a  working 
unit.  By  using  a  communications  standard,  the  vehicle 
will  have  inherent  interoperability  with  other 
autonomous  systems.  This  will  allow  future 
experimentation  with  other  autonomous  solution 
developers. 

All  this  allows  for  the  vehicle  to  be  developed  with  a 
current  focus  on  dismounted  following  using  vision- 
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based  techniques  without  pigeonholing  the  final  product  so  that  it  cannot  be  expanded 
upon  and  used  for  future  research  efforts. 

Vehicle 

ISAAC  was  designed  to  test  advancements  in  dismounted 
leader-follower  technologies.  To  do  this  it  would  need  to 
go  where  a  soldier  can  go  and  move  like  a  soldier  would 
move  -  and  do  all  this  on  a  limited  budget.  To  this  end,  the 
vehicle  itself  is  based  on  two  Bombardier  ATVs  spliced 
together;  giving  it  dual- Ackerman  steering  and  a  ready¬ 
made  suspension.  By  going  this  route,  a  majority  of  the 
vehicle  testing  was  eliminated  and  the  designers  need  only  to  focus  on  the  new  subset  of 
dynamics  created  with  the  merger.  The  robotic  and  vehicular  controls  are  housed  in 
multiple  NEMA  electrical  enclosures  mounted  strategically  on  the  robot’s  chassis.  The 
original  vehicle  power  system  was  left  intact  and  an  additional  3000  Watt  generator  was 
integrated  into  the  system  to  supply  the  controls  package. 

Subsystem 

The  High-Level  Controller  is  a  ruggedized  rack  system 
consisting  of  three  1.8  GHz  Pentium-M  class  processors  with 
1GB  of  DDR  RAM,  60Gig  Hard  Drives,  and  10/100  Ethernet 
connections.  It  has  a  16  port  10/100  Switch,  three  IEEE  1394 
(Firewire)  ports,  and  a  dedicated  power  supply.  Two  of  the 
processors  mn  Win2k  Professional,  while  the  third  runs  Linux. 
The  first  two  processors  deal  with  the  interface  to  the  Low-Level  Controller  and  Path 
Planning,  while  the  Linux  box  is  dedicated  to  vision  processing.  This  is  more  than 
enough  processing  power  to  keep  the  update  refresh  rate  under  5  seconds. 

Primitive 

The  Low-Level  Controller  is  a  conglomeration  of  multiple  dedicated  processors  tied 
together  by  an  industrial  PLC.  This  allows  for  the  segmenting  of  the  various  tasks 
(steering,  braking,  transmission,  throttle,  and  choke)  while  continuously  monitoring  for 
safety  and  control.  Steering  is  done  using  a  SmartMotor  package  that  consists  of  a  DC 
rotary  motor,  amplified,  and  controller  in  one  complete  package.  The  brake -by- wire 
system  was  developed  by  Delphi  and  is  a  modified  version  of  one  of  their  commercially 
available  products.  The  transmission  is  controlled  by  a  field-effect  linear  actuator 
system.  This  not  only  gives  precise  control,  which  is  necessary  when  gears  stick,  but  also 
allows  for  the  manual  shifting  required  during  catastrophic  failure.  The  throttle  and 
choke  are  controlled  by  the  same  system  using  high-torque  programmable  digital  quarter 
scale  servos.  The  PLC,  that  links  this  altogether,  is  a  57series  Premium  processor  from 
Telemecanique.  It  has  supports  multiple  styles  of  programming,  has  nearly  unlimited 
Input/Output  capability  (due  to  a  distributed  IO  network),  and  supports  multiple 
communication  networks  including;  CAN,  RS232,  RS485,  and  802. 1  lg.  All  of  these  fall 
under  the  500ms  requirement  for  tltis  level. 

Servo 
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Aside  from  the  fundamental  IO  components  (limit  switched,  pressure  switches,  etc. . .) 
required  for  vehicle  operation  the  only  sensors  currently  installed  on  the  vehicle  are  the 
Stereo  Pair  used  for  pedestrian  following.  The  pedestrian  following  system  uses  2 
MicroPix  C640  cameras  to  capture  stereo  imagery.  They  have  a  24-bit  RGB  color  output 
and  can  capture  30  frames  per  second  at  640x480.  The  cameras  can  be  replaced  by 
several  other  types  of  IIDC  firewire  cameras  with  similar  output.  These  cameras  are 
placed  at  least  two  feet  apart  and  aligned  so  that  each  cameras  horizontal  field  of  view  is 
on  the  same  plan  with  the  other. 

I 

Pedestrian  Following 

Following  a  pedestrian  will  be  based  off  of  a  sequence  of  images.  Initialization  at  the 
start  of  the  system  requires  a  user  to  select  the  pedestrian  that  the  robotic  vehicle  should 
follow.  After  this,  the  image  of  the  pedestrian  is  used  as  the  initial  template  matching  of 
the  next  image.  The  image  template  of  the  pedestrian  is  processed  using  color  blob 
analysis  and  disparity.  Dynamic  template  matching  is  used  to  combat  the  changes  in  the 
followed  person’s  image.  This  is  accomplished  by  replacing  the  last  template  with  the 
new  found  isolated  person.  Changes  in  intensity  are  combated  by  isolating  the  area 
around  the  last  location  of  the  person  in  the  new  image  and  normalizing  it.  This  makes 
the  last  image  template  the  same  intensity  as  the  new  image. 

The  tracking  and  following  of  the  system  will  be  based  off  of  a  3-D  world  coordinate 
system.  The  reason  for  this  is  so  the  pedestrian  following  system  can  be  placed  on  any 
robotic  system,  with  minor  modifications,  that  uses  GPS  following.  The  only  major 
discrepancy  is  that  this  system  will  consider  the  current  location  as  0,0,0  and  the  new 
calculated  position,  where  the  followed  person  current  is  x,y,z.  Updates  will  take  place 
every  second  or  so  with  a  buffer  so  the  robotic  vehicle  does  not  run  into  the  person. 

State-of-the-bot 

So  where  along  are  we  in  this  endeavor?  Mechanically,  the  vehicle  is  about  90% 
complete.  It  is  currently  getting  final  modifications  applied.  These  would  be  the 
mounting  of  the  3kW  generator,  the  steering  encoders,  and  front  and  rear  bumpers.  We 
still  need  to  mount  the  vision  system  and  we’ll  need  to  calibrate  the  brake-by-wire  system 
and  bleed  the  lines. 

Electrically,  the  vehicle  is  not  as  far  along.  The  high-level  controller  is  in  the  final  stages 
of  testing.  We  recently  worked  out  a  problem  of  random  resets  when  all  four  boards  (3 
CPU  +  Ethernet  cards)  are  running  at  the  same  time.  The  problem  was  with  the  power 
supply.  After  talking  to  the  board  vendor,  it  was  discovered  that  the  boards  would  reset  if 
power  dropped  below  4.8  VDC  level.  We  didn't  have  a  supply  that  had  sense  lines  to 
allow  for  feedback  to  the  supply  to  keep  the  voltage  level  at  the  back  plane  to  5  V.  The 
only  way  around  this  was  to  adjust  the  voltage  to  a  level  to  not  drop  below  4.9  volts  at  the 
back  plane.  This  approach  appears  to  have  mitigated  the  problem.  If  this  issue  becomes 
a  problem  in  the  future,  we  will  have  to  acquire  a  power  supply  with  sense  feedback 
capabilities.  Currently,  all  of  the  low-level  controllers  are  running  together  on  a  test-rack 
in  our  lab.  When  the  vehicle  gets  back  from  the  final  mechanical  modifications,  we’ll 
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begin  the  process  of  transferring  equipment  from  the  test-rack  to  the  vehicle’s  control 
cabinets. 

The  pedestrian  detection  and  tracking  system  is  in  the  beginning  stages.  The  design  of 
the  system  is  complete  and  the  algorithms  are  starting  to  be  debugged  in  Matlab.  Test 
images  are  taken,  but  a  more  thorough  set  of  images  need  to  be  taken.  The  design  of  the 
code  is  underway.  The  preliminary  testing  of  the  system  should  take  place  in  June  with  a 
more  completed  system  testing  on  ISAAC  in  August. 

The  wav  ahead 

So  where  are  we  going  with  this  project?  We  are  currently  working  towards  an  initial 
demo  in  mid  to  late  May  of  2005.  This  will  be  a  sub-system  level  test  where  we  prove 
out  the  individual  technologies  and  verify  they  all  communicate  together.  The  goal  is  to 
tele -op  the  research  platform. 

Later  in  the  year  around  early  August,  we  plan  to  demo  our  initial  following  capability. 
Two  modes  of  operation  are  being  investigated.  The  first  is  direct  following,  where  the 
vehicle  tracks  directly  toward  the  leader.  This  is  the  simplest  form  of  vision-based 
following  and  is  therefore  a  great  starting  point.  It  is  this  mode  we  have  targeted  for  the 
August  demo. 

After  we’ve  established  Direct  Following,  we’ll  develop  Exact  Following.  This  is  a 
mode  of  operation  where  the  vehicle  follows  the  exact  path  the  leader  takes.  The  vehicle 
will  track  the  leader  wherever  they  move  in  the  field  of  view  and  than  follow  this  path. 
This  will  allow  the  vehicle  to  take  advantage  of  the  leaders  pathing  decisions  and  allow  it 
to  overcome/avoid  obstacles  in  the  same  manor  as  the  leader.  This  gives  the  vehicle  a 
better  mobility  capability  without  requiring  advanced  obstacle  detection.  We  have  yet  to 
set  a  milestone  for  this  demo. 


Looking  to  the  Future 

ISAAC  and  our  vision-based  following  research  won’t  come  to  an  end  with  the 
completion  of  our  August  demo.  We’re  speculating  that  it  will  take  a  minimum  of  six  to 
eight  months  to  refine  the  Exact  Following  mode  and  after  that  we’ve  already  got  plans 
for  the  next  phases  of  our  research. 

One  of  the  logical  extensions  of  vision-based  following  is  gesture-based  control. 
Controlling  an  unmanned  ground  vehicle  by  using  simple  gestures  would  add  yet  another 
venue  to  the  user  interface  community.  By  developing  a  system  that  uses  the  standard 
military  ground  guide  commands  any  soldier  could  direct  any  such  equipped  UGV  in  the 
depot  without  the  need  of  a  specific  soldier  machine  interface. 

Another  field  of  interest  is  integrating  the  work  being  developed  by  our  Hands-Free  Tele- 
Operation  via  Physiological  Signal  Recognition  SBIR.  This  involves  using  a  microphone 
in  the  ear  canal  (similar  to  a  hearing  aid)  to  detect  sub-vocal  and  tongue-position 
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commands.  The  current  system  looks  like  it  will  be  easily  adapted  to  our  PDA  user 
interface  and  could  make  for  some  really  exciting  experimentation. 

On  top  of  all  this  a  few  other  groups  have  expressed  interest  in  using  the  ISAAC  platform 
to  test  their  research  efforts.  This  highlights  one  of  the  biggest  advantages  we  see  in 
having  our  in-house  developed  research  platform  -  we  built  it  so  we  know  exactly  how  it 
works.  We  can  add  and  remove  anything  we  so  desire  without  worrying  about 
proprietary  technology  getting  in  our  way. 


Who  Else  Can  Benefit  from  This? 

Although  our  current  applications  are  inline  with  the  FCS  vision  of  the  MULE  and 
dismounted  following,  the  technology  itself  is  not  limited  to  military  robotics.  Any 
scenario  where  you  need  to  carry  a  heavy  load  could  be  so  enhanced.  Consider  a  golf 
caddy  that  follows  you  from  hole  to  hole,  or  a  blacktop/gravel  hauler  for  city  street  repair 
keeping  step  with  its  work  force.  These  are  just  a  few  of  the  many  examples  that  could 
come  to  mind. 


ATD  Advanced  Technology  Demonstrator 

ATV  All  Terrain  Vehicle 

CAN  Component  Area  Network 

CMU  Carnegie  Mellon  University 

COTS  Commercial  Off  The  Shelf 

CPU  Central  Processing  Unit 

DDR  Double  Data  Rate  [RAM] 

DGPS  Differential  GPS 

FCS  Future  Combat  System 

GPS  Global  Position  System 

IEEE  Institute  for  Electronic  &  Electrical  Engineers 

ILIR  In-house  Laboratory  Independent  Research 

ISAAC  Intelligent  Systems  And  Autonomous  Controls 

JAUS  Joint  Architecture  for  Unmanned  Systems 

MULE  Multifunction  Utility/Logistics  Equipment  Vehicle 

NEMA  National  Electrical  Manufacturer’s  Association 

OE  Operating  Environment 

OFW  Objective  Force  Warrior 

ORD  Operational  Requirements  Document 

PLC  Programmable  Logic  Contioller 

RDECOM  Research,  Development,  and  Engineering  Command 

SBIR  Small  Business  Innovation  Research 

SUGV  Soldier  (or  Small)  Unmanned  Ground  Vehicle 

TARDEC  Tank-automotive  and  Armaments  Research  Development  and  Engineering  Center 

UGV  Unmanned  Ground  Vehicle 


1. )  Global  Security  Organization:  http://www.globalsecuritv.org/militarv/svstcms/around/fcs.htm 

2. )  Performance  Specification  for  MULE,  ORD  for  FCS 

3. )  Christian  Science  Monitor:  http://csmonitor.com/2005/0127/pl4s02-stct.htm 
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